Skip to content

Conversation

@vishesh92
Copy link
Member

@vishesh92 vishesh92 commented Jan 15, 2026

Description

This PR fixes #11564

Initially the scheduler relied on the database constraints to ensure no duplicates and catch the exception and just continue. But the way persist method is implemented, it will always logs the error for SQLIntegrityConstraintViolationException. In this PR, we add a check before creating a new scheduled job.

Generated Summary

This pull request adds a safeguard to prevent duplicate VM scheduled jobs by checking for existing jobs before creating new ones. It introduces a new DAO method to efficiently find jobs by schedule and timestamp, and updates the scheduler logic to use this method. Additionally, it introduces some minor import changes.

Database access improvements:

  • Added a new method findByScheduleAndTimestamp(long scheduleId, Date scheduledTimestamp) to the VMScheduledJobDao interface and its implementation, along with the necessary SearchBuilder in VMScheduledJobDaoImpl, to efficiently look up scheduled jobs by schedule ID and timestamp. [1] [2] [3] [4]

Scheduler logic enhancements:

  • Updated VMSchedulerImpl.scheduleNextJob to use the new DAO method to check for an existing scheduled job before creating a new one, preventing duplicate scheduling for the same VM and time.

Code maintenance:

  • Added missing imports for SQLException and SQLIntegrityConstraintViolationException in VMSchedulerImpl.java.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@vishesh92 vishesh92 requested a review from Copilot January 15, 2026 06:50
@vishesh92 vishesh92 linked an issue Jan 15, 2026 that may be closed by this pull request
@vishesh92 vishesh92 force-pushed the vmscheduler-remove-redundant-logs branch from 99ab23a to bc5f3b9 Compare January 15, 2026 06:55
@vishesh92 vishesh92 marked this pull request as ready for review January 15, 2026 06:56
@vishesh92 vishesh92 force-pushed the vmscheduler-remove-redundant-logs branch from bc5f3b9 to f4f07f6 Compare January 15, 2026 06:57
@vishesh92 vishesh92 requested review from Copilot and removed request for Copilot January 15, 2026 06:57
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request improves the VM scheduler to prevent duplicate job scheduling and reduce redundant exception logging. The changes introduce a proactive check for existing scheduled jobs before attempting to create new ones, addressing issue #11564.

Changes:

  • Added a new DAO method findByScheduleAndTimestamp to check for existing scheduled jobs
  • Updated scheduler logic to query for existing jobs before attempting to insert, reducing exception occurrences
  • Removed the unused exception variable from the catch clause to clean up the code

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
VMScheduledJobDao.java Added interface method for finding jobs by schedule ID and timestamp
VMScheduledJobDaoImpl.java Implemented the new query method using SearchBuilder pattern
VMSchedulerImpl.java Added pre-insert check for duplicate jobs and removed exception variable name
Comments suppressed due to low confidence (3)

server/src/main/java/org/apache/cloudstack/vm/schedule/VMSchedulerImpl.java:51

  • These imports for SQLException and SQLIntegrityConstraintViolationException are not used anywhere in the code. They should be removed to keep the imports clean.
import java.time.ZonedDateTime;
import java.util.ArrayList;

server/src/main/java/org/apache/cloudstack/vm/schedule/VMSchedulerImpl.java:175

  • There is a potential race condition between the check for an existing job at line 167 and the persist operation at line 175. If two threads call scheduleNextJob concurrently with the same schedule and timestamp, both could pass the null check and attempt to insert, causing one to throw EntityExistsException. While the exception is caught, this race condition could still result in unnecessary database operations and exception handling. Consider using a database-level check or a synchronized block around the check-and-insert operation.
            logger.trace("Job is already scheduled for schedule {} at {}", vmSchedule, scheduledDateTime);
            return scheduledDateTime;
        }

        scheduledJob = new VMScheduledJobVO(vmSchedule.getVmId(), vmSchedule.getId(), vmSchedule.getAction(), scheduledDateTime);
        try {
            vmScheduledJobDao.persist(scheduledJob);
            ActionEventUtils.onScheduledActionEvent(User.UID_SYSTEM, vm.getAccountId(), actionEventMap.get(vmSchedule.getAction()),
                    String.format("Scheduled action (%s) [vm: %s, schedule: %s] at %s", vmSchedule.getAction(), vm, vmSchedule, scheduledDateTime),

server/src/main/java/org/apache/cloudstack/vm/schedule/VMSchedulerImpl.java:181

  • The existing test testScheduleNextJobScheduleScheduleExists does not cover the new code path introduced by findByScheduleAndTimestamp. The test should be updated to mock vmScheduledJobDao.findByScheduleAndTimestamp to return null first (for the new check) and then have persist throw EntityExistsException. Additionally, a new test should be added to verify the scenario where findByScheduleAndTimestamp returns an existing job, ensuring the method returns early without attempting to persist.
            logger.trace("Job is already scheduled for schedule {} at {}", vmSchedule, scheduledDateTime);
            return scheduledDateTime;
        }

        scheduledJob = new VMScheduledJobVO(vmSchedule.getVmId(), vmSchedule.getId(), vmSchedule.getAction(), scheduledDateTime);
        try {
            vmScheduledJobDao.persist(scheduledJob);
            ActionEventUtils.onScheduledActionEvent(User.UID_SYSTEM, vm.getAccountId(), actionEventMap.get(vmSchedule.getAction()),
                    String.format("Scheduled action (%s) [vm: %s, schedule: %s] at %s", vmSchedule.getAction(), vm, vmSchedule, scheduledDateTime),
                    vm.getId(), ApiCommandResourceType.VirtualMachine.toString(), true, 0);
        } catch (EntityExistsException exception) {
            logger.debug("Job is already scheduled.");
        }
        return scheduledDateTime;
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codecov
Copy link

codecov bot commented Jan 15, 2026

Codecov Report

❌ Patch coverage is 13.33333% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.23%. Comparing base (5c1f931) to head (f4f07f6).
⚠️ Report is 15 commits behind head on 4.20.

Files with missing lines Patch % Lines
...udstack/vm/schedule/dao/VMScheduledJobDaoImpl.java 0.00% 10 Missing ⚠️
...apache/cloudstack/vm/schedule/VMSchedulerImpl.java 40.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               4.20   #12428   +/-   ##
=========================================
  Coverage     16.23%   16.23%           
  Complexity    13380    13380           
=========================================
  Files          5657     5657           
  Lines        498996   499010   +14     
  Branches      60566    60567    +1     
=========================================
+ Hits          81011    81030   +19     
+ Misses       408951   408943    -8     
- Partials       9034     9037    +3     
Flag Coverage Δ
uitests 4.03% <ø> (ø)
unittests 17.09% <13.33%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Contributor

@DaanHoogland DaanHoogland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@DaanHoogland DaanHoogland added this to the 4.20.3 milestone Jan 15, 2026
@RosiKyu RosiKyu self-assigned this Jan 19, 2026
@RosiKyu
Copy link
Collaborator

RosiKyu commented Jan 20, 2026

@blueorangutan package

@blueorangutan
Copy link

@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Copy link
Contributor

@sureshanaparti sureshanaparti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code lgtm

@sureshanaparti sureshanaparti changed the title Remove redundant Exceptions from logs for vm schedules Remove redundant Exceptions from logs for VM schedules Jan 20, 2026
@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16450

Copy link
Collaborator

@RosiKyu RosiKyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Summary

Tested on both single-MS and multi-MS (3 nodes) environments. The PR successfully eliminates SQLIntegrityConstraintViolationException errors caused by race conditions when multiple management servers attempt to schedule the same VM job simultaneously.

Test Environment Result
Baseline Check Single-MS Pass
TC1: Setup - Create Network and Deploy VM Single-MS Pass
TC2: Create VM Schedules Single-MS Pass
TC3: Verify Scheduled Jobs in Database Single-MS Pass
TC4: Monitor Logs for Errors Single-MS Pass
TC5: Schedule Execution and Database Integrity Single-MS Pass
Multi-MS TC1: Environment Setup and Baseline 3-MS Pass
Multi-MS TC2: Create VM and Schedules 3-MS Pass
Multi-MS TC3: Concurrent Scheduler Execution (Critical) 3-MS Pass
Multi-MS TC4: Zero Errors After Concurrent Execution 3-MS Pass
Multi-MS TC5: Database Integrity - No Duplicates 3-MS Pass

Single Management Server Testing

Environment

  • CloudStack Version: 4.20.3.0-SNAPSHOT (with PR #12428)
  • Management Servers: 1 node (10.0.35.182)
  • KVM Hosts: 2
  • Zone: ref-trl-10681-k-Mol9-rositsa-kyuchukova (Advanced networking)

Baseline Check

Objective: Verify no pre-existing SQLIntegrityConstraintViolationException errors in logs before testing

Expected Result 0 or a baseline count to compare against

Actual Result 0 errors in logs

Test Evidence

[root@ref-trl-10681-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# grep -c "SQLIntegrityConstraintViolationException" /var/log/cloudstack/management/management-server.log
0
[root@ref-trl-10681-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk
Apache CloudStack 🐵 CloudMonkey 6.5.0
Report issues: https://github.com/apache/cloudstack-cloudmonkey/issues

(localcloud) 🐱 > list virtualmachines filter=name,id,state,
{
  "count": 1,
  "virtualmachine": [
    {
      "id": "516abe4f-f1d4-4543-bfec-6094ac28155e",
      "name": "vm-schedule-test",
      "state": "Running"
    }
  ]
}

TC1: Setup - Create Network and Deploy VM

Objective: Create isolated network and deploy test VM for schedule testing

Expected Result: Network created, VM deployed in Running state

Actual Result: Network ID: 4bbdf8d5-f9ba-4f8e-8efe-66717780e9c1, VM ID: 516abe4f-f1d4-4543-bfec-6094ac28155e, State: Running

Test Evidence:

(localcloud) 🐱 > list virtualmachines filter=name,id,state
{
  "count": 1,
  "virtualmachine": [
    {
      "id": "516abe4f-f1d4-4543-bfec-6094ac28155e",
      "name": "vm-schedule-test",
      "state": "Running"
    }
  ]
}

TC2: Create VM Schedules

Objective: Create STOP and START schedules for the test VM

Expected Result: Both schedules created and enabled

Actual Result: STOP schedule ID: 63e84437-ff1a-4077-b2fb-07305a00ffcd, START schedule ID: 4bb897a6-8c80-4f9a-9610-569accc60804, both enabled

Test Evidence:

(localcloud) 🐱 > list vmschedule virtualmachineid=516abe4f-f1d4-4543-bfec-6094ac28155e 
{
  "count": 2,
  "vmschedule": [
    {
      "action": "START",
      "enabled": true,
      "id": "4bb897a6-8c80-4f9a-9610-569accc60804",
      "schedule": "2-59/5 * * * *"
    },
    {
      "action": "STOP",
      "enabled": true,
      "id": "63e84437-ff1a-4077-b2fb-07305a00ffcd",
      "schedule": "*/5 * * * *"
    }
  ]
}

TC3: Verify Scheduled Jobs Created in Database

Objective: Confirm scheduled jobs are created correctly in vm_scheduled_job table

Expected Result: Jobs created for both STOP and START schedules with correct timestamps

Actual Result: 2 jobs created - STOP at 10:10:00 UTC, START at 10:12:00 UTC

Test Evidence:

[root@ref-trl-10681-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# mysql -u root -p -e "SELECT id, vm_schedule_id, action, scheduled_timestamp FROM cloud.vm_scheduled_job ORDER BY scheduled_timestamp DESC LIMIT 10;"
+----+----------------+--------+---------------------+
| id | vm_schedule_id | action | scheduled_timestamp |
+----+----------------+--------+---------------------+
|  2 |              2 | START  | 2026-01-22 10:12:00 |
|  1 |              1 | STOP   | 2026-01-22 10:10:00 |
+----+----------------+--------+---------------------+

TC4: Monitor Logs for Errors (Critical Test)

Objective: Verify no SQLIntegrityConstraintViolationException errors occur during schedule execution

Expected Result: 0 errors, clean scheduler execution in logs

Actual Result: 0 SQLIntegrityConstraintViolationException errors. Scheduler executed cleanly.

Test Evidence:

[root@ref-trl-10681-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# grep -c "SQLIntegrityConstraintViolationException" /var/log/cloudstack/management/management-server.log
0
  • STOP schedule execution at 10:10:
2026-01-22 10:10:24,545 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2317c721]) (logid:9575d01a) VM scheduler.poll is being called at 2026-01-22T10:10:00+0000
2026-01-22 10:10:24,587 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2317c721]) (logid:9575d01a) Got 1 scheduled jobs to be executed at 2026-01-22T10:10:00+0000
2026-01-22 10:10:24,588 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2317c721]) (logid:9575d01a) Executing STOP for VM VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"516abe4f-f1d4-4543-bfec-6094ac28155e"} for scheduled job: VMScheduledJob {"action":"STOP","asyncJobId":null,"id":1,"uuid":"bb24844a-7cbc-4c4d-9145-0a224afe3108","vmId":3,"vmScheduleId":1} at 2026-01-22T10:10:00+0000
2026-01-22 10:11:00,146 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:[ctx-8bcf3a24, job-38, ctx-5b907e01]) (logid:6d02d159) Complete async job-38, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"516abe4f-f1d4-4543-bfec-6094ac28155e","name":"vm-schedule-test",...,"state":"Stopped",...}
  • START schedule execution at 10:12:
2026-01-22 10:12:24,545 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-83f2d2ac]) (logid:12cd0bf0) VM scheduler.poll is being called at 2026-01-22T10:12:00+0000
2026-01-22 10:12:24,561 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-83f2d2ac]) (logid:12cd0bf0) Got 1 scheduled jobs to be executed at 2026-01-22T10:12:00+0000
2026-01-22 10:12:24,562 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-83f2d2ac]) (logid:12cd0bf0) Executing START for VM VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Stopped","type":"User","uuid":"516abe4f-f1d4-4543-bfec-6094ac28155e"} for scheduled job: VMScheduledJob {"action":"START","asyncJobId":null,"id":2,"uuid":"353e5b8c-1d3a-4875-83d0-9fe8483f1227","vmId":3,"vmScheduleId":2} at 2026-01-22T10:12:00+0000
2026-01-22 10:12:30,840 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-14eea5ee, job-40, ctx-5dab88b7]) (logid:46aff9f2) Complete async job-40, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"516abe4f-f1d4-4543-bfec-6094ac28155e","name":"vm-schedule-test",...,"state":"Running",...}

TC5: Verify Schedule Execution and Database Integrity

Objective: Confirm scheduled jobs execute correctly and no duplicate entries exist

Expected Result: Jobs have async_job_id populated after execution, no duplicate entries

Actual Result: Both jobs executed successfully (async_job_id 38 and 40), no duplicates, next jobs scheduled

Test Evidence:

[root@ref-trl-10681-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# mysql -u root -p -e "SELECT id, vm_schedule_id, action, scheduled_timestamp, async_job_id FROM cloud.vm_scheduled_job ORDER BY scheduled_timestamp DESC LIMIT 10;"
+----+----------------+--------+---------------------+--------------+
| id | vm_schedule_id | action | scheduled_timestamp | async_job_id |
+----+----------------+--------+---------------------+--------------+
|  4 |              2 | START  | 2026-01-22 10:17:00 |         NULL |
|  3 |              1 | STOP   | 2026-01-22 10:15:00 |         NULL |
|  2 |              2 | START  | 2026-01-22 10:12:00 |           40 |
|  1 |              1 | STOP   | 2026-01-22 10:10:00 |           38 |
+----+----------------+--------+---------------------+--------------+

[root@ref-trl-10681-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# mysql -u root -p -e "SELECT vm_schedule_id, scheduled_timestamp, COUNT(*) as cnt FROM cloud.vm_scheduled_job GROUP BY vm_schedule_id, scheduled_timestamp HAVING cnt > 1;"
Empty set (0.00 sec)

Multi-Management Server Testing

Environment

  • CloudStack Version: 4.20.3.0-SNAPSHOT (with PR #12428)
  • Management Servers: 3 nodes
    • mgmt1: 10.0.32.153
    • mgmt2: 10.0.32.154
    • mgmt3: 10.0.34.16
  • KVM Hosts: 2
  • Zone: ref-trl-10687-k-Mol9-rositsa-kyuchukova (Advanced networking)

Multi-MS TC1: Environment Setup and Baseline

Objective: Verify 3-MS cluster is operational and baseline has no errors

Expected Result: All 3 MS nodes Up, 0 pre-existing errors

Actual Result: All 3 MS nodes Up, 0 errors on all nodes

Test Evidence:

mysql> SELECT id, name, msid, state, removed FROM cloud.mshost WHERE removed IS NULL;
+----+-------------------------------------------------------------------+----------------+-------+---------+
| id | name                                                              | msid           | state | removed |
+----+-------------------------------------------------------------------+----------------+-------+---------+
|  1 | ref-trl-10687-k-mol9-rositsa-kyuchukova-mgmt1.sofia.shapeblue.com | 32986003145004 | Up    | NULL    |
|  2 | ref-trl-10687-k-mol9-rositsa-kyuchukova-mgmt2.sofia.shapeblue.com | 32988704276781 | Up    | NULL    |
|  3 | ref-trl-10687-k-mol9-rositsa-kyuchukova-mgmt3.sofia.shapeblue.com | 32986103808675 | Up    | NULL    |
+----+-------------------------------------------------------------------+----------------+-------+---------+

[root@mgmt1 ~]# grep -c "SQLIntegrityConstraintViolationException" /var/log/cloudstack/management/management-server.log
0
[root@mgmt2 ~]# grep -c "SQLIntegrityConstraintViolationException" /var/log/cloudstack/management/management-server.log
0
[root@mgmt3 ~]# grep -c "SQLIntegrityConstraintViolationException" /var/log/cloudstack/management/management-server.log
0

Multi-MS TC2: Create VM and Schedules

Objective: Deploy test VM and create START/STOP schedules

Expected Result: VM Running, both schedules created and enabled

Actual Result: VM 366b3cb0-6b4e-4714-a7fd-49e465ebcc24 Running, STOP and START schedules created

Test Evidence:

(localcloud) 🐱 > list virtualmachines filter=name,id,state
{
  "count": 1,
  "virtualmachine": [
    {
      "id": "366b3cb0-6b4e-4714-a7fd-49e465ebcc24",
      "name": "vm-schedule-test",
      "state": "Running"
    }
  ]
}

(localcloud) 🐱 > list vmschedule virtualmachineid=366b3cb0-6b4e-4714-a7fd-49e465ebcc24
{
  "count": 2,
  "vmschedule": [
    {
      "action": "START",
      "enabled": true,
      "id": "1a87b9d3-0dfc-42a2-89a4-d93a194bd64d",
      "schedule": "2-59/5 * * * *"
    },
    {
      "action": "STOP",
      "enabled": true,
      "id": "2bd2a7b1-f5f6-4355-a89c-deef20d2d896",
      "schedule": "*/5 * * * *"
    }
  ]
}

Multi-MS TC3: Verify Scheduler Runs on All 3 MS Nodes Simultaneously (Critical Race Condition Test)

Objective: Confirm all 3 MS nodes run VMSchedulerPollTask concurrently without SQLIntegrityConstraintViolationException errors

Expected Result: All 3 MS pick up same scheduled job, no duplicate entry errors

Actual Result: All 3 MS ran scheduler simultaneously at 10:42:00 UTC, 0 errors on all nodes

Test Evidence:

  • mgmt1 (10.0.32.153):
2026-01-22 10:42:05,764 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-17794a3a]) (logid:258a45bd) VM scheduler.poll is being called at 2026-01-22T10:42:00+0000
2026-01-22 10:42:05,769 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-17794a3a]) (logid:258a45bd) Got 1 scheduled jobs to be executed at 2026-01-22T10:42:00+0000
2026-01-22 10:42:05,770 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-17794a3a]) (logid:258a45bd) Executing START for VM VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"366b3cb0-6b4e-4714-a7fd-49e465ebcc24"}
2026-01-22 10:42:05,789 WARN  [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-17794a3a]) (logid:258a45bd) Skipping action (START) for [vm: VM instance...] because VM is in state: Running
  • mgmt2 (10.0.32.154):
2026-01-22 10:41:56,870 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cc606b24]) (logid:c2400d14) VM scheduler.poll is being called at 2026-01-22T10:42:00+0000
2026-01-22 10:41:56,881 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cc606b24]) (logid:c2400d14) Got 1 scheduled jobs to be executed at 2026-01-22T10:42:00+0000
2026-01-22 10:41:56,882 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cc606b24]) (logid:c2400d14) Executing START for VM VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"366b3cb0-6b4e-4714-a7fd-49e465ebcc24"}
2026-01-22 10:41:56,902 WARN  [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cc606b24]) (logid:c2400d14) Skipping action (START) for [vm: VM instance...] because VM is in state: Running
  • mgmt3 (10.0.34.16):
2026-01-22 10:41:50,967 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-165f8579]) (logid:823227c7) VM scheduler.poll is being called at 2026-01-22T10:42:00+0000
2026-01-22 10:41:50,995 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-165f8579]) (logid:823227c7) Got 1 scheduled jobs to be executed at 2026-01-22T10:42:00+0000
2026-01-22 10:41:50,997 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-165f8579]) (logid:823227c7) Executing START for VM VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"366b3cb0-6b4e-4714-a7fd-49e465ebcc24"}
2026-01-22 10:41:51,012 WARN  [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-165f8579]) (logid:823227c7) Skipping action (START) for [vm: VM instance...] because VM is in state: Running

Multi-MS TC4: Verify Zero Errors After Concurrent Execution

Objective: Confirm 0 SQLIntegrityConstraintViolationException errors across all MS nodes after multiple schedule cycles

Expected Result: 0 errors on all 3 MS nodes

Actual Result: 0 errors on all 3 MS nodes

Test Evidence:

[root@mgmt1 ~]# grep -c "SQLIntegrityConstraintViolationException" /var/log/cloudstack/management/management-server.log
0
[root@mgmt2 ~]# grep -c "SQLIntegrityConstraintViolationException" /var/log/cloudstack/management/management-server.log
0
[root@mgmt3 ~]# grep -c "SQLIntegrityConstraintViolationException" /var/log/cloudstack/management/management-server.log
0

Multi-MS TC5: Database Integrity - No Duplicate Entries

Objective: Verify no duplicate job entries exist in vm_scheduled_job table

Expected Result: No duplicate entries, jobs executed with async_job_id populated

Actual Result: No duplicates, jobs executed successfully

Test Evidence:

mysql> SELECT id, vm_schedule_id, action, scheduled_timestamp, async_job_id FROM cloud.vm_scheduled_job ORDER BY scheduled_timestamp DESC LIMIT 10;
+----+----------------+--------+---------------------+--------------+
| id | vm_schedule_id | action | scheduled_timestamp | async_job_id |
+----+----------------+--------+---------------------+--------------+
|  6 |              1 | STOP   | 2026-01-22 10:55:00 |         NULL |
|  5 |              2 | START  | 2026-01-22 10:52:00 |         NULL |
|  4 |              1 | STOP   | 2026-01-22 10:50:00 |           44 |
|  3 |              2 | START  | 2026-01-22 10:47:00 |           42 |
|  1 |              1 | STOP   | 2026-01-22 10:45:00 |           40 |
|  2 |              2 | START  | 2026-01-22 10:42:00 |         NULL |
+----+----------------+--------+---------------------+--------------+

mysql> SELECT vm_schedule_id, scheduled_timestamp, COUNT(*) as cnt FROM cloud.vm_scheduled_job GROUP BY vm_schedule_id, scheduled_timestamp HAVING cnt > 1;
Empty set (0.00 sec)

@DaanHoogland DaanHoogland merged commit d1eb282 into apache:4.20 Jan 22, 2026
30 of 32 checks passed
@DaanHoogland DaanHoogland deleted the vmscheduler-remove-redundant-logs branch January 22, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recurrent insert errors in database on vm schedules

5 participants